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D EVICE FOR THE D EIEgH ON Of A OTRUCTUnC^O R F A PPLIED TO A . 
_ SUBSTRATE. AND SUITABLE PERTINENT METHODS 



RELATED APPLICATION 

The present application relates to and claims priority from PCT/EP2003/012354 
filed November 5, 2003, titled "DEVICE FOR THE DETECTION OF A STRUCTURE 
TO BE APPLIED TO A SUBSTRATE AND SUITABLE PERTINENT METHODS", 
the complete subject matter of which is hereby expressly incorporated in its entirety. 

BACKGROUND OF INVENTION 

[0001] The present invention generally relates to a device for the detection of a 
structure to be applied to a substrate, as well as suitable pertinent methods. 

[0002] It has been conventional to perform optical measurements in order to de- 
tect a structure to be applied to a substrate, whereby often various systems for fully- 
automatic inspection of the structure, including adhesive and sealing agent extrusion 
Unes, have been used. For this purpose, one or multiple video-cameras are trained on the 
structure to be detected. In addition, an illumination module is required whose purpose it 
is to generate a camera image that is rich in contrast. The inspection of the structure is 
performed in a delayed fashion, several seconds after application of the structure to the 
substrate. In many cases, the inspection is not performed until all of the structure is ap- 
plied to the substrate. This is disadvantageous in that the inspection is performed separate 
and independent of the process of application, which may be tedious and difficult to han- 
dle in some of the cases. Hitherto, these systems were not stabile enough and to tedious 
in their parameterization to allow direct inspection. 



SUMMARY OF INVENTION 

[0003] In certain embodiments, a device is provided for the detection of a 
structure to be applied to a substrate and suitable pertinent methods such that, on the one 
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hand, direct inspection of the structure applied is feasible and, on the other hand, inspec- 
tion is easy to perform. 

[0004] Moreover, a device and method are provided for the detection of a struc- 
ture to be applied to a substrate, including for subsequent inspection, such that, on the 
one hand, subsequent inspection is feasible in a simple fashion, and, on the other hand, an 
accurate error analysis for the structure to be applied is provided. 

[0005] A sensor unit is provided on the facility for the application of the struc- 
ture. By this means, a visual inspection system with a compact design is provided, 
whereby the illumination module can preferably also be provided on the facility for the 
application of the structure. This facilitates the integration of the device according to the 
present application into existing systems whose task it is to apply a structure to a sub- 
strate. While the structure is applied to the substrate, if an error is present, it is feasible 
to directly act or interrupt during the manufacturing process and/or sort out the defective 
substrate. This provides for improved efficiency in the manufacture of structures on a 
substrate. If the method involves a tested area of the structure that is placed along the 
structure to be tested by means of support points, the handling becomes trouble-fi-ee 
since the interactive process between the user and the displayed structure is implemented 
in a simple fashion with currently existing means. If, according to the invention, the 
range of tolerance is set along the reference line defined by the support points, inaccura- 
cies of the structure, if any, will be accounted for and, in particular, the quality inspection 
of the structure to be tested can be set individually by this means. This simplified opera- 
tor interaction allows even complex track profiles of the structure to perform a teach-in 
process in a simple and efficient fashion. Moreover, the existing display visualizing the 
structure and the reference line generated by the support points indicates directly to the 
user whether or not deviations in the track profile of the structure are present. 

[0006] Further advantageous embodiments are the subject matter of the 
dependent claims. 

[0007] By positioning the sensor unit directly at the exit of the facility for the 
application of the structure, it becomes feasible to provide a compact and highly- 
integrated implementation of the device. Therefore, the sensor unit is capable of ftilly 
automatic high-speed inspection of the structure almost directly after its application. The 
sensor unit comprises a video-sensor, and may use various image detection procedures. 
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The video-sensor may comprise one and/or several picture lines, (e.g., 15 lines), such that 
a high image recording rate can be achieved. By this means the device stays small in size 
and the image analysis can be performed in the sensor unit such that no external data 
analysis facility is needed. 

[0008] The use of a white light illumination module as illumination module al- 
lows the use of conventional halogen lamps also for the generation of white light. The 
use of an LED illumination module as illumination module allows for the provision sen- 
sor illumination for improved contrast between background and structure by skillfully 
combining different spectral ranges. The analysis as such can therefore proceed in a 
stabile fashion and the resource use involved in the analytical logics is minimized also. 
The same applies in particular to the provision of multiple illumination modules, which 
can therefore provide for improved contrast. If, in addition, the analytical unit is inte- 
grated in the sensor unit, it becomes easy to add to the device the feature of setting the 
quality criteria in a simple fashion by means of an extemal control unit. The transmission 
preferably is mediated by radio, infrared data or cable. 

[0009] If the method used involves that the structure is determined by means of 
so-called calipers (gray edge scanning), which preferably extend orthogonal to the struc- 
ture on the substrate, this means can be used to define specific areas, preferably crossing 
areas, between the caliper line and a contrast structure in the area to be determined. If the 
calipers extend orthogonal to the structure on the substrate, this allows especially the 
width of the structure to be determined in a simple fashion. In conjunction with appropri- 
ate visualization software, the profile of the structure and the corresponding areas of error 
can be displayed. The user thus recognizes immediately whether or not the profile of the 
structure complies with the given range of tolerance or if the structure is being applied 
inaccurately. Another advantage is provided by makihg it feasible to base the structure 
determination and corresponding error analysis for example on the given substrate data, 
such as recesses and elevations, since this allows more exact statements concerning the 
profile of the structure to be made. 

[0010] It has proven to be advantageous to base the determination of structure 
on the analysis of the brightness profiles of the gray values along the caliper. Therefore, 
the gray values can be used to determine in which place an area to be determined is to be 
subjected to structure inspection; in particular it becomes feasible to determine a position, 
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at which the change from object to background is the highest. This is achieved by using 
the second derivative of the gray value profile for structure detection. The values to be 
determined are determined exactly as sub-pixels. If a set of hypotheses is generated for 
each caliper, especially for the case of four nodes of the caliper, a set of six options of 
variation is obtained, which each differ by the distance of the positions of the individual 
nodes of the caliper. 

[0011] By linking neighboring sets of hypotheses to each other, certain values 
can be assigned, especially through the use of a heuristic function, on the basis of which 
the relevant nodes for the edge of the structxire can be determined. 

[0012] Further advantageous refinements are the subject matter of the remain- 
ing dependent claims. 
BRIEF DESCRIPTION OF DRAWINGS 

[0013] In the following, advantageous refinements of the invention shall be il- 
lustrated on the basis of the following drawings. 

[0014] Figure 1 shows schematically an advantageous embodiment of the de- 
vice according to the invention. 

[0015] Figure 2 shows a sub-area of the structure applied in Figure 1. 
[0016] Figure 3 shows an error analysis. 

[0017] Figure 4 shows the application of the calipers to an area to be defined, 
which contains both the structure and deviations. 

[0018] Figure 5 shows the crossing points of the relevant contrast lines and the 

caliper. 

[0019] Figure 6 shows the generation of a set of hypotheses from a caliper. 
[0020] Figure 7 shows the structure determination from neighboring sets of hy- 
potheses. 

[0021] Figure 8 shows the method for the determination and/or elimination of 

deviated edges and/or determination of the structure. 

DETAILED DESCRIPTION OF EMBODIMENTS 

[0022] Figure 1 shows a device 1 for the application of a structure 9, such as an 

adhesive extrusion line, to a substrate 7. The position of device 1 is adjustable in x, y, 
and z direction. Optionally, the device may be fixed in position and the substrate may be 
adjustable in x, y, and z direction. The device 1 further comprises a sensor unit 3 (e.g., a 
video sensor), which, in this embodiment, is positioned directly at the exit of the device 1 
for the application of the structure. Also shown in this schematic drawing is the illumina- 
tion module 5, which provides for the contrast during the application and/or registration 
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of the areas to be monitored. It can be seen in this embodiment that a so-called adhesive 
extrusion line 9 is being applied to and/or introduced into a pre-made recess 13 in the 
substrate 7. Reference number 1 1 shows by shaded lines an area of the image shown in 
more detail in Figure 2. 

[0023] The device 1 includes an analytic unit 6 (e.g., processor) that communi- 
cates with the sensor imit 3 and memory 2. The sensor unit 3 obtains video or still im- 
ages of the area 11 of the substrate 7 and line 9. The images are stored in memory 2 
and/or displayed on monitor 12 (e.g., a 3D display). The analytic xmit 6 determines sets 
of data from the images, where the sets of data correspond to the stmcture 9 on the sub- 
strate 7. The analytic unit 6 may be controlled by an external control unit 14 (e.g., an 
infrared data transmission), used to set quality criteria. The analytic unit 6 may include 
visualization software. 

[0024] Optionally, a network connection 15 may provide trigger and analysis 
over the intemet or an intranet. 

[0025] Figure 2 shows, for example, the recess 13 into which the structure 
and/or adhesive extrusion line 9 is introduced. This selected area can be processed in the 
analytical unit 6 in the sensor unit 3, or it can, as a matter of principle, be displayed to 
the user right during the application process such that the user can manually set his sup- 
port points 20 on the basis of which a reference line 22 can be generated. As is clearly 
evident from Figure 2, a range of tolerance is defined with regard to the reference line 22, 
which approximately reflects the course of the structure, which range of tolerance in this 
case is equidistant to the reference line. Accordingly, it is being tested whether or not the 
reference line defined by the support points is within the range of tolerance. In addition to 
the range of tolerance. Figure 2 shows an inspection area 26, in which the structure is 
situated. 

[0026] Figure 3 shows an error display, for example, which does not only iden- 
tify the position of the error in the application of the structure, but also indicates the mag- 
nitude of the error to the user based on the analytical accuracy of the method according to 
the invention. The user can then decide on the basis of the magnitude of the error whether 
or not the deviation from the set value is tolerable or if the manufacturing process needs 
to be terminated. Accordingly, the method allows to make a decision on the basis of 
direct inspection of the application of the structure in the course of the manufacturing 
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process, in a fully automatic fashion, as to whether or not the manufacturing process 
needs to be interrupted and/or if the defective substrate needs to be sorted out. 

[0027] The analytical procedure is described in the following by Figures 4 to 
8. Figure 4 shows the so-called edge extraction of the features present in the inspection 
area. For this purpose, a set of calipers, which preferably extend orthogonal to the track 
of the structure, is placed over the inspection area, whereby the extraction of the edges 
thus proceeds orthogonal to the track of the structure due to the analysis of the brightness 
profile of the gray values. This determines a position reflecting the change firom object to 
background, in which this change is most pronounced. This is achieved by calculating the 
second derivative of the profile of the gray values. The values to be determined are thus 
determined at sub-pixel accuracy. 

[0028] Figure 5 shows the tracing of the stmcture's track after edge extraction, 
whereby all edges found for each line by means of the node points are shown. 

[0029] Figure 6 shows that a set of hypotheses is generated for each caliper of 
Figures 4 and 5, whereby, for example, for four node points of a caliper a total of six po- 
sition hypotheses are exist. Subsequently, the caliper hypotheses are gradually, preferably 
in a hierarchical fashion, linked to the corresponding neighbor and/or neighboring sets of 
hypotheses. This linkage is performed in an iterative fashion, as shown in Figure 7. For 
this purpose, left and right hypotheses are generated progressively, which in turn are 
linked to each other and/or analyzed using a heuristic fimction. One selection criterion for 
defining the determination of structure can, for example, be *the higher the value deter- 
mined, the better is the underlying hypothesis'. 

[0030] Figure 8 clearly illustrates how the iterative procedure of the individual 
sets of hypotheses is applied. In the process, for example the sets of hypotheses, 2, 3, 4 in 
Figure 6 (I-II, I-III, II-III, II-II) are linked in a combinatorial fashion, whereby in each 
case the left hypothesis of hypothesis 3 is linked to the corresponding right hypothesis. 
This in turn results in an assignment of the hypotheses, whereby a value is determined on 
the basis of the heuristic function. Because of the pre-determined rule, according to 
which "the higher the value, the better is the hypothesis", the structure can then be deter- 
mined by eliminating the hypotheses with a lower value according to the heuristic func- 
tion if the number of hypotheses thus developed exceeds the permissible number of hy- 
potheses per existing node. 
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[0031] These methods can be used to determine the structure precisely and ac- 
curately and with few sets of data such that direct a determination of the structure, for 
example during the application of the structure, is feasible. It should be noted in this con- 
text that the heuristic function uses the following criteria to determine the set value. 

1 . Level of edge contrast 

2. Width of structure 

3. Difference between set vs actual position 

4. Co-linea.rity of the actual position 

5. Difference between set vs actual width of the stmcture 

6. Co-linearity of the actual width of the structure 

7. Difference between set vs actual brightness of the structure 

8. Co-linearity of the actual brightness of the structure 

9. Difference between set vs actual brightness of the background 

1 0. Co-linearity of the actual brightness of the background 

[0032] Based on the actual implementation, according to which the device is 
used during the application of an adhesive extrusion line to a substrate, it is advantageous 
to comply with the following: according to an advantageous embodiment, the system 
and/or device according to the invention consists essentially of a color line video-sensor 
with an integral analytical unit and illumination for imaging and illumination of the seal- 
ing agent and/or adhesive extrusion line. The components reside in a compact protective 
housing. The visual inspection system is attached directly downstream from the adhesive 
application system (application nozzle) and is trained on the area shortly downstream 
from the adhesive nozzle in order to perform a test directly after the application of the 
extrusion line. The test is therefore performed directly after the application of the sealing 
agent or adhesive allowing the quality of the extrusion line to be analyzed (for breaks, 
position and placement, thickness) while it is being applied. 

[0033] A video-sensor records only one or several picture lines (maximally 15 
lines) in order to achieve a high image recording rate. The analysis is performed in the 
color line video-sensor with an integral analytical unit. An external data analysis facility 
(analytical PC) is not required, since the video-sensor itself includes a miniaturized ana- 
lytical computer. The quality criteria (lO/NIO limit values) are set by means of an exter- 
nal control unit connected to the sensor via a radio connection, infrared data transmission 
connection (IrDa) or cable connection (serial or network). 
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[0034] Depending on the surface properties of the adhesive and/or sealing 
agent, one / several 

• white light illumination module(s), e.g. halogen lamp(s), 
and/or 

• . LED illumination module(s) with various colors are used to illumi- 
nate the track of the adhesive structure. 

[0035] The illumination modules are compact in design to allow them to be in- 
stalled in a compact system (image recording sensor and illumination in a joint housing). 
- For this purpose, provision are made for combining various different illumination mod- 
ules (differing in stmctural shape, color) in order to achieve high contrast between back- 
ground and adhesive by suitably combining different spectral ranges of illumination and 
sensor. Accordingly, the analysis can proceed in a stabile fashion and the resource use 
required for the analytical logics can be kept low. The illumination module contains a 
white light illumination module. The illumination module may include a LED illumina- 
tion module radiating the spectral ranges, red, blue, green, infrared and/or ultraviolet. 

[0036] The purpose of the visualization software is to display errors made dur- 
ing the application of extrusion lines of adhesive. For this purpose, the adhesive track to 
be traced is stored as a 3D track and the corresponding error areas are marked therein. 
The corresponding errors are highlighted through the use of a different color and labeled 
with additional text. The software and/or sensor communicates with a robot or any other 
control unit using any of the conMnon field buses (Profibus, Interbus, Devicenet), 
Ethernet, serial interface, OPC - server or any other available communication interfaces. 
In the offline version, the robot track is programmed and stored ahead of time. After the 
process of adhesive application, the visuaUzation software can be triggered and then ob- 
tains the respective error areas from the robot. In the online version, the visualization 
software is provided at all times during the run with the current position along the robot's 
track and, if there is an error, with an error code. 

[0037] In addition, data can be accepted from CAD files. The data of the com- 
ponent contained therein, i.e. the adhesive track or similar data, can be co-processed and 
displayed jointly with the corresponding error sites in a 3-dimensional or 3-dimensional 
display. 
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[0038] In order to simplify the user interaction, a GUI special-developed for the 
inspection of adhesive tracks was used. Simple mouse clicks can be used to enter com- 
plex track profiles in a simple and efficient fashion. The graphical elements are designed 
such that the set limit values, such as min / max ranges and range of tolerance are easy to 
see (Figure 2). Changes in the track of the profile can also be made with just a few mouse 
clicks. In this context, there is no need to enter the adhesive track exactly, since the 
downstream image processing operations are sufficiently stabile to compensate for the 
inaccuracies generated during input of the information. An additional display provides 
the operator with information concerning any production errors. By clicking on an error 
with the mouse, the respective area is enlarged and the plain text description of the error 
is displayed (Figure 3). 

[0039] The mathematical linkage shovra in the following is used to determine 
the heuristic function for the structure determination, i.e. a heuristic value for elementary 
hypotheses and a heuristic value for complex hypotheses. 

Heuristic value for elementary Iiypotheses 

The following applies to an input vector: 

^ ~ {^weighn^-^weight2^^pos\9^pos29^br9^bk]* 

wherein: 

^weight} weight of the first point, 
^weight2 weight of the second point, 
^pos\ position of the first point, 
^pos2 position of the second point, 

brightness of the structure, 
x^f^ brightness of the background. 

The following applies to the set values: 

^ ~ {^width9^br9^bk} 

wherein: 

s width set width, 

s^^ set brightness of the structure, 
Sf^f^ set brightness of the background. 
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with the heuristic coefficients: 

^ = {^consif ^weight >^pos>^wUllh'^bry^bk\ 



The heuristic value, h, takes the following form: 
wherein: 



width 



^pos2 ^pos\ ^ width 



= abs ( 



Heuristic value for complex hypotheses 

The following applies to an input vector: 

^ ~ {^Ipos^^lwidthi-^lbr^-^lbk^^rpos^-^nvidth^^rbr^^rbklf 



wherein: 



ipos 



Vwidth 



^Ibr 



^Ibk 



rpos 



^rwidth 



\br 



^rbk 



position on the right side of the left hypothesis, 

width on the right side of the left hypothesis, 

brightness of the structure on the right side of the left hypothesis, 

brightness of the background on the right side of the left hypothesis, 

position on the left side of the right hypothesis, 

width on the left side of the right hypothesis, 

brightness of the structure on the left side of the right hypothesis, 

brightness of the background on the left side of the right hypothesis. 



with the heuristic coefficients: 
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b = {bpos^t^width^tf^r^i^bk} 

The heuristic value, A, takes the following form: 
wherein: 

^ width ^ ip^lwidth ~ ^rwidth \ 

^br ^(^b^{^lbr-^rbr\ 

and 

h^^ji heuristic value of left hypothesis 
^right heuristic value of right hypothesis 



